Papers with MT metrics
Difficulty-Aware Machine Translation Evaluation (2021.acl-short)
Copied to clipboard
| Challenge: | Current MT evaluation measures pay the same attention to each sentence component . in real-world examinations, the questions vary in difficulty and weightings . |
| Approach: | They propose a difficulty-aware MT evaluation metric that takes translation difficulty into account . they propose to use this metric to evaluate machine translation (MT) results . |
| Outcome: | The proposed method outperforms most MT evaluation metrics in terms of human correlation. |
EffEval: A Comprehensive Evaluation of Efficiency for MT Evaluation Metrics (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a recent surge of interest in developing evaluation metrics based on pretrained large language models (LLMs) can better cope with lexical variation. |
| Approach: | They propose to replace computation-intensive transformers with lighter alternatives and employ linear and quadratic approximations for alignment algorithms on top of LLM representations. |
| Outcome: | The proposed approach replaces computation-intensive transformers with lighter alternatives and employs linear and quadratic approximations for alignment algorithms on top of LLM representations. |
Understanding the Properties of Minimum Bayes Risk Decoding in Neural Machine Translation (2021.acl-long)
Copied to clipboard
| Challenge: | Neural Machine Translation (NMT) currently exhibits biases such as producing translations that are too short and overgenerating frequent words. |
| Approach: | They propose to use minimum bayes risk decoding instead of beam search to investigate the effects of beam decoding on unbiased samples. |
| Outcome: | The proposed method improves on a number of previously reported biases and failure cases of beam search on unbiased samples. |
Part Represents Whole: Improving the Evaluation of Machine Translation System Using Entropy Enhanced Metrics (2022.findings-aacl)
Copied to clipboard
| Challenge: | Existing machine translation metrics have poor correlations with human assessments . entropy-based evaluations are often limited to a limited number of samples . |
| Approach: | They propose a fast and unsupervised approach to enhance machine translation metrics using entropy by introducing sentence-level difficulty. |
| Outcome: | The proposed method outperforms existing metrics on five sub-tracks in the WMT19 Metrics shared tasks. |
Has Machine Translation Evaluation Achieved Human Parity? The Human Reference and the Limits of Progress (2025.acl-short)
Copied to clipboard
| Challenge: | In machine translation evaluation, metric performance is assessed based on agreement with human judgments. |
| Approach: | They incorporate human baselines into the MT meta-evaluation to gain a clearer understanding of metric performance and establish an upper bound. |
| Outcome: | The results suggest human parity, but there are several reasons to caution . |
Breeding Machine Translations: Evolutionary approach to survive and thrive in the world of automated evaluation (2023.acl-long)
Copied to clipboard
| Challenge: | a genetic algorithm (GA) based method improves MT quality and identifies weaknesses in evaluation metrics. |
| Approach: | They propose a genetic algorithm-based method for modifying n-best lists produced by a machine translation system using a fitness function. |
| Outcome: | The proposed method improves translation quality and identifies weaknesses in evaluation metrics. |
Beyond Literal Mapping: Benchmarking and Improving Non-Literal Translation Evaluation (2026.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have advanced machine translation (MT) a meta-evaluation dataset focused on non-literal translations is lacking . experimental results show the inaccuracies of traditional MT metrics and the limitations of LLM-as-a-Judge. |
| Approach: | They propose a meta-evaluation framework that leverages sub-agents to evaluate machine translation metrics. |
| Outcome: | The proposed framework improves on the knowledge cutoff and score inconsistency problem. |
Beyond Reference: Evaluating High Quality Translations Better than Human References (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing machine translation metrics give maximum score to reference sentence . however, these metrics overlook the possibility that candidate sentences outperform reference sentences in terms of quality. |
| Approach: | They propose a machine translation metrics that give an absolute score to a translated sentence based on the similarity with the reference sentence. |
| Outcome: | The proposed measure outperforms existing MT metrics in terms of quality and assigns positive scores to candidates that outperformed reference sentences. |
Remedy-R: Generative Reasoning for Machine Translation Evaluation without Error Annotations (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent MT metrics like xCOMET, Met-ricX, and Remedy have strong correlations with human preferences, but they are black boxes, revealing little insight into why a translation is good or bad. |
| Approach: | They propose a reasoning-driven generative MT metric trained with reinforcement learning from pairwise translation preferences without requiring error-span annotations or distillation from closed LLMs. |
| Outcome: | The proposed reasoning-driven generative MT metric produces step-by-step analyses of accuracy, fluency, and completeness, enabling more interpretable assessments. |
Evaluating Automatic Subtitling: Correlating Post-editing Effort and Automatic Metrics (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing metrics for automatic subtitling are not yet fully explored. |
| Approach: | They propose to use machine translation metrics to measure post-editing effort in automatic subtitling to collect data on product-, process- and participant-based data. |
| Outcome: | The proposed metrics correlate with measures of post-editing effort in automatic subtitling. |
Extrinsic Evaluation of Machine Translation Metrics (2023.acl-long)
Copied to clipboard
| Challenge: | MT metrics are widely used to distinguish the quality of machine translation systems across relatively large test sets. |
| Approach: | They evaluate the segment-level performance of the most widely used MT metrics by correlating them with how useful they are for downstream tasks. |
| Outcome: | The MT metrics are widely used to distinguish the quality of machine translation systems across relatively large test sets. |
Prompting PaLM for Translation: Assessing Strategies and Performance (2023.acl-long)
Copied to clipboard
| Challenge: | Large language models trained on multilingual but not parallel text exhibit remarkable ability to translate between languages. |
| Approach: | They investigate the pathways language model which has demonstrated the strongest machine translation performance among similarly-trained LLMs to date. |
| Outcome: | The pathways language model (PaLM) has demonstrated the strongest machine translation performance among similarly-trained LLMs to date. |
Beyond Correlation: Interpretable Evaluation of Machine Translation Metrics (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have shown that MT metrics return assessments as scalar scores that are difficult to interpret, posing a challenge to making informed design choices. |
| Approach: | They propose an interpretable evaluation framework that evaluates MT metrics in two scenarios that serve as proxies for filtering and translation re-ranking use cases. |
| Outcome: | The proposed framework offers clearer insights than correlation with human judgments. |
Exploring Context Strategies in LLMs for Discourse-Aware Machine Translation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models excel at machine translation, but the impact of how LLMs utilize different forms of contextual information on discourse-level phenomena remains underexplored. |
| Approach: | They examine how different forms of context influence standard MT metrics and specific discourse phenomena such as formality, pronoun selection, and lexical cohesion. |
| Outcome: | Evaluating multiple LLMs across multiple domains and language pairs, the findings consistently show that context boosts translation and discourse-specific performance. |